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IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, 
STORAGE MEDIUM, AND PROGRAM 

BACKGROUND OF THE INVENTION 

5 

Field of the Invention 

[0001] The present invention relates to an image 
processing device, image processing method, storage medium, 
and program for correcting images . 

10 

Description of the Related Art 

[0002] Various methods have been disclosed, which relate 
to an algorithm for correcting contrast, highlight, shadow, 
or white balance of a digital camera image. 

15 [0003] Japanese Patent Laid-Open No. 11-317959 (US Patent 

No. 6,493, 468) by the same inventor as this application 
discloses a method in which image features are determined by 
the ratio of uneven distribution of color components in a 
highlight portion of the image, and in which, based on the 

20 determination result, the degree of white balance correction, 

contrast, highlight, and shadow are optimally corrected to 
reproduce the color of an object. 

[0004] In the above-described conventional example, 
however, the image features are determined from histograms 
25 of an overall image, and a correction is performed by using 
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one and the same correction method irrespective of whether 
the scene is of a person's image or a landscape image. In 
general, the desired degree of contrast enhancement is 
different between personal images and landscape images. 

5 

SUMMARY OF THE INVENTION 



[0005] Accordingly, it is an object of the present 
invention to solve the above-described problem, and to 
10 provide an image processing device capable of changing a 

correction method in response to whether a specific object 
is included in an input image. 

[0006] It is another object of the present invention to 
provide an image correcting unit that, when the specific 
15 object is a person, performs a corresponding image 

correction . 

[0007] In order to achieve these objects, the present 
invention provides an image processing device that includes 
a detector for detecting whether an input image includes a 

20 specific object; a setting unit for setting a correction 

parameter based on the above-described detection result; and 
a correcting unit for correcting the input image by using 
the set correction parameter. 

In a further aspect, an image processing method is 

25 disclosed, which comprises the steps of creating a histogram 
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of an input image, and then using the histogram to set a 
first gradation correcting condition. The first gradation 
correcting condition is then used to correct the input image, 
after which, a specific object region is extracted from the 
5 corrected input image. 

The image processing method further includes the 
steps of creating a histogram of the extracted specific 
object region, and then using this histogram to set a second 
gradation correcting condition. Thereafter, the image in 
10 the specific object region is corrected by using the set 

second gradation correcting condition. 

[0008] Other features and advantages of the present 
invention will be apparent from the following description 
taken in conjunction with the accompanying drawings, in 
15 which like reference characters designate the same or 

similar parts throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

20 [0009] Fig. 1 is a block diagram showing the construction 

of an image processing device according to a first 
embodiment of the present invention. 

[0010] Fig. 2 is a flowchart showing image processing in 
the first embodiment. 
25 [0011] Fig. 3 is a flowchart showing operational process 
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steps associated with an image input section 2 in the first 
embodiment . 

[0012] Fig. 4 is a flowchart showing a processing flow of 
image correction in the first embodiment. 
5 [0013] Fig. 5 is a flowchart showing operational process 

steps associated with a histogram creating section 4 in the 
first embodiment. 

[0014] Fig. 6 is a representation of an example of a 
histogram in the first embodiment. 
10 [0015] Fig. 7 is a flowchart showing operational process 

steps associated with a lookup table creating section 6 in 
the first embodiment. 

[0016] Figs. 8A and 8B are diagrams showing examples of 
lookup tables held in a lookup table holding section 7 in 

15 the first embodiment. 

[0017] Fig. 9 is a diagram showing an example of an 
output object profile in the first embodiment. 
[0018] Fig. 10 is a diagram showing an example of an 
input device profile in the first embodiment. 

20 [0019] Fig. 11 is a diagram showing an example of an 

output device profile in the first embodiment. 
[0020] Fig. 12 is a flowchart showing operational process 
steps in an image correcting section 8 in the first 
embodiment . 

25 [0021] Fig. 13 is a flowchart showing operational process 
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steps in a face detection processing section 12 in the first 
embodiment . 

[0022] Fig. 14 is a flowchart showing a flow of image 
correction processing when using mask data in the first 
5 embodiment . 

[0023] Fig. 15 is a flowchart showing operational process 
steps associated with the histogram creating section 4 when 
using mask data in the first embodiment. 

[0024] Fig. 16 is a flowchart showing operational process 
10 steps in the lookup table creating section 6 when using mask 

data in the first embodiment. 

[0025] Figs. 17A and 17B are diagrams showing examples of 
lookup tables held in the lookup table holding section 7 
when using mask data in the first embodiment. 
15 [0026] Fig. 18 is a flowchart showing operational process 

steps in a file processing section 11 in the first 
embodiment . 

[0027] Fig. 19 is a flowchart showing a processing flow 
of edge enhancement in the first embodiment. 
20 [0028] Fig. 20 is a table showing examples of 

photographic information and image data stored in an image 
buffer 9 in the first embodiment. 

[0029] Fig. 21 is a flowchart showing a method for 
synthesizing a lookup table for correcting gradation 
25 reproducibility and color reproducibility in the first 
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embodiment . 

[0030] Fig. 22 is a diagram showing examples of the 
synthesized lookup tables for correcting gradation 
reproducibility and color reproducibility in the first 
5 embodiment . 

[0031] Figs. 23A to 23D are representations of 
relationship between the posture of a digital camera during 
photographing and the data developing method in the image 
buffer 9. 

10 [0032] Figs. 24A to 24C are representations of a 

processing flow of face detection. 
[0033] Fig. 25 is a block diagram showing the 
construction of an image processing device according to a 
second embodiment of the present invention. 

15 [0034] Fig. 26 is a table showing information held in a 

corrected information holding section 15 in the second 
embodiment . 

[0035] Fig. 27 is a flowchart showing a processing flow 
of creation of lookup table for exposure correction in a 
20 third embodiment of the present invention. 

[0036] Fig. 28 is a diagram showing the relationship 
among the representative luminance, the target luminance, 
and the desirable luminance range of a face in the third 
embodiment . 



25 



- 7 - 



DESCRIPTION OF THE PREFERRED EMBODIMENTS 
First Embodiment 

[0037] Fig. 1 shows a block diagram of the main section 
5 of an image processing device according to a first 

embodiment of the present invention. 

[0038] Referring to Fig. 1, this image processing device 
comprises an image input section 2, image output section 3, 
histogram creating section 4, histogram holding section 5, 

10 lookup table creating section 6, lookup table holding 

section 7, image correcting section 8, imager buffer 9, 
image buffer 10 for use in copying, filter processing 
section 11, face detection processing section 12, and 
photographic information holding section 13. Here, the 

15 image input section 2 reads data from an input image 1, and 

writes it into the image buffer 9. 

[0039] The image output section 3 writes data stored in 
the image buffer 9 into an output image 14. The histogram 
creating section 4 creates a histogram based on image data 
20 stored in the image buffer 9, and stores the created result 

in the histogram holding section 5. 

[0040] The histogram holding section 5 holds a histogram 
of the overall image data and a histogram of a mask region. 
[0041] The lookup table creating section 6 calculates 
25 parameters necessary for correction, based on the histogram 
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stored in histogram holding section 5, and creates a lookup 
table. In addition, the lookup table creating section 6 
synthesizes the aforementioned lookup table and a lookup 
table for correcting gradation characteristic and color 
5 reproducibility, which is held in advance in the lookup 

table holding section 7. Then, the lookup table creating 
section 6 stores the synthesized lookup table in the lookup 
table holding section 7. 

[0042] Thus, the lookup table holding section 7 holds the 
10 lookup tables created in the lookup table creating section 6 

and the lookup table for correcting gradation characteristic 
and color reproducibility. 

[0043] The image correcting section 8 corrects the image 
stored in the image buffer 9, based on the lookup table 
15 stored in the lookup table holding section 7, and then 

stores the corrected result in the image buffer 10 for use 
in copying. 

[0044] The image buffer 9 and the image buffer 10 are 
used for copying and holding image data. 
20 [0045] The filter processing section 11 performs 

enlargement/reduction processing, and edge enhancement in a 
highlight portion. 

[0046] The face detection processing section 12 detects 
whether specific objects, such as a human face is present. 
25 This detection may be based on the image data stored in the 
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image buffer 9. If a human face is detected, the face 
detection processing section 12 then creates mask data. 
[0047] The photographic information holding section 13 
holds various types of information such as the camera 
5 posture during photography. Other information includes the 

usage or non-usage of a flash, focal length, and in-focus 
position (two dimensional), and the like. 

[0048] The image processing device can be implemented by, 
for example, supplying a personal computer with a program 
10 for achieving processing or control with respect to each of 

the sections shown in Fig. 1. Then, CPU of the personal 
computer executes each processing or control, based on 
contents of the program. 

[0049] Hereinafter, more detailed descriptions of this 
15 embodiment will be made using a concrete example. 

Processing Flow in This Embodiment 

[0050] Fig. 2 shows processing flow in this embodiment. 

[0051] At step SI, in the image input section 2, 
photographic information and image data of the input image 1 
20 are read, and are developed in the image buffer 9. 

Operational process steps in the image input section 2 are 
shown in Fig. 3, and are discussed later. 

[0052] At step S2, in the histogram creating section 4, 
lookup table creating section 6, and image correcting 
25 section 8, an image is corrected. Operational steps in 
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these sections are shown in Fig. 4, and are discussed later. 
[0053] At step S3, in the face detection processing 
section 12, face detection processing is performed with 
respect to the face as the specific object to be detected. 
5 Operational process steps in the face detection processing 
section 12 are shown in Fig. 13, and are discussed later. 
[0054] At step S4, it is determined whether any faces are 
presented, based on the result of the face detection 
processing. If at least one face is present, the processing 
10 advances to step S5. If no face is present, the processing 

advances to step S6. 

[0055] At step S5, in the histogram creating section 4, 
lookup table creating section 6, and image correcting 
section 8, the image is corrected by using mask data. 
15 Operational process steps in these sections are shown in Fig. 
14, and are discussed later. 

[0056] At step S6, in the filter processing section 11, 
filter processing is performed. Operational process steps 
in the filter processing section 11 are shown in Fig. 18, 

20 and are discussed later. 

[0057] At step S7, in the image output section 3, the 
image data and photographic information stored in the image 
buffer 9 are outputted to the output image 14. 
Photographic Information and Image Data Stored in Input 

25 Image 1 
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[0058] Fig. 20 shows photographic information and image 
data stored in the input image 1. The photographic 
information is stored in the photographic information 
holding section 13, and the image data is stored in the 
5 image buffer 9. The image buffer 9 stores mask data in 

addition to the image data. The photographic information 
includes information about the posture of a camera (Fig. 23A 
shows its normal position; four different postures are shown 
in Figs. 23A to 23D) , the usage or non-usage of a flash, 
10 focal lengths, and in-focus positions, an input device 

(camera) ID and the like. 
Method for Developing Image 

[0059] Fig. 3 shows operational process steps associated 
with the image input section 2. These are more specific 

15 descriptions of step SI shown in Fig. 2. 

[0060] At step Sll, photographic information is inputted 
from the input image 1, and is stored in the photographic 
information holding section 13. An example of photographic 
information is information specified by the known Exif 

20 standard. However, any other information that constitutes 

information about photographing by a camera or information 
about image inputting by a scanner, may be used. 
[0061] At step S12, the posture of a camera during 
photographing that is stored in the photographic information 

25 holding section 13 is determined, and in accordance with the 
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determination result, the image data obtained from the input 
image 1 is developed in the image buffer 9. 

[0062] When the posture of the camera is as shown in Fig. 
23A, the image data is developed in the image buffer as it 
5 is. When its posture is as Fig. 23B, the image data is 

developed after it is rotated 90° to the left; when its 
posture is as shown in Fig. 23C, the image data is developed 
after a 90° rotation to the right; and when its posture is 
as Fig. 23D, the image data is developed after a 180° 

10 rotation, whereby the image data is developed in the image 

buffer so that an up-and-down direction conforms to that in 
the case shown in Fig. 23A. Thereby, when performing later 
face detection processing, it is possible to treat a person 
as being always at a normal position irrespective of the 

15 posture of the camera during photography. 

[0063] During development, processing for reducing dark 
noise peculiar to a digital camera is also performed. This 
processing may include a treatment such as smoothing an 
image in, for example, portions where the luminance thereof 

20 is low and delicately fluctuates. 

Image Correction Processing 

[0064] Fig. 4 shows a flow of image correction processing. 
[0065] At step 21, in the histogram creating section 4, a 
histogram is created based on the image data stored in the 
25 image buffer 9, and the obtained result is stored in the 
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histogram holding section 5. Operational process steps in 
the histogram creating section 4 are shown in Fig. 5, and 
are discussed later. 

[0066] At step 22, in the lookup table creating section 6, 
5 a lookup table is created. Operational process steps in the 
lookup table creating section 6 are shown in Fig. 7, and are 
discussed later. 

[0067] At step 23, in the image correcting section 8, the 
image is corrected. Operational process steps in the image 
10 correcting section 8 are shown in Fig. 12, and are discussed 

later . 

Operations in Histogram Creating Section 4 

[0068] Fig. 5 shows operational process steps in the 

histogram creating section 4. These are more specific 

15 descriptions of step S21 shown in Fig. 4. 

[0069] At step 31, one pixel of image data is fetched one 
after another from the image buffer 9. The luminance (R, G, 
and B values) of respective R(red), G(green), and B(blue) 
colors are then stored as image data. At step S32, from R, 

20 G, and B luminance values of the image data, the luminance L 

of the pertinent pixel is determined by the following 
expression . 

[0070] L = (3*R + 6*G + 1*B) /10 

At step S33, the histogram stored in the histogram holding 
25 section 5 is updated. The histogram holding section 5 holds 
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a histogram HistL having the luminance L, calculated above, 
and HistR, HistG, and HistB that store the cumulative 
luminance values of respective R, G, and B colors by 
luminance L of the pertinent pixel. Any initial state is 0. 
5 The following expressions are used to perform histogram 

updates . 

[0071] HistR [L] = HistR [L] + R 
HistG [L] = HistG [L] + G 
HistB [L] = HistB [L] + B 

10 HistL [L] = HistL [L] + 1 

At step S34, it is checked whether all pixels have been 
completed. Otherwise, the processing advances to step S31. 
Here, a histogram may be created by using color information 
about some pixels sampled at an appropriate pitch from all 

15 pixels, instead of using color information about all pixels. 

[0072] Fig. 6 shows an example of a histogram HistL. 

Method for Creating Lookup Table for Correcting Gradation 
Reproducibility and Color Reproducibility 

20 [0073] The lookup table for correcting gradation 

reproducibility and color reproducibility refers to a lookup 
table that, with a view toward achieving a desirable color 
reproduction, performs an image correction such that the 
gradation reproduction and color reproduction on printed 

25 matter when an image inputted from an input device such as a 
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digital camera is outputted by an output device such as a 
printer, is brought near to the gradation reproduction and 
color reproduction implemented by an output object such as a 
silver salt positive film. A method for creating a lookup 
5 table for correcting gradation reproducibility and color 

reproducibility is explained with reference to Fig. 21. 
[0074] At step S121, an output object profile (the 
relationship between the patch number and the XYZ 
colorimetric value) shown in Fig. 9, and an output device 

10 profile (the relationship between the XYZ colorimetric 

values and the R, G, and B luminances) shown in Fig. 11 are 
synthesized. The synthesis method is firstly to search for 
the patch having the same Y value as the Y value of the XYZ 
values (a value is provided every patch) of the output 

15 object profile out of the output device profile. Then, the 

R, G, and B values corresponding to the XYZ values of the 
patch searched from the output device profile, and the patch 
number of the original output object profile are brought 
into correspondence with each other, and they are 

20 synthesized. The synthesized table represents the 

relationship between the patch number and the luminances (R, 
G, and B values) of the output device. 

[0075] At step S122, the synthesized table in step S121 
and an input device profile shown in Fig. 10 are synthesized. 
25 Because the input device profile represents the relationship 
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between the patch number and the luminances (R, G, and B 
values) of the input device, the synthesis method is to 
acquire the relationship between the luminances (R, G and B 
values) of the input device and those of the output device 
5 corresponding to the identical patch number. The created 
table is stored in the lookup table holding section 7, and 
is used when correcting the image. An example of how it is 
actually used is described later. 

10 Examples of Lookup Tables for Correcting Gradation 

Characteristic and Color Reproducibility 

[0076] Lookup tables shown in Fig. 22 are lookup tables 
LUTRDD, LUTGDD, and LUTBDD for correcting gradation 
characteristic and color reproducibility that have been 

15 created by the above-described method. Intermediate points 
that are not stored in the table are interpolated by a 
straight line, and thereby conversion data is created. 
Points in the vicinity of the opposite ends are interpolated 
using straight lines between respective points (255, 255) 

20 and (0, 0) . In this embodiment, linear interpolation is 

used as interpolation processing. Alternatively, for 
example, nonlinear interpolation processing employing, for 
example, a spline curve or a Bezier curve may be used. 
Operations in Lookup Table Creating Section 6 

25 [0077] Operational process steps in the lookup table 
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creating section 6 are shown in Fig. 7. These are more 
specific descriptions about step S22 shown in Fig. 4. 
[0078] At step S41, the maximum luminance of the image is 
determined from the histogram HistL stored in the histogram 
5 holding section 5. In the histogram shown in Fig. 6, the 

maximum luminance is 252. 

[0079] At step S42, the maximum luminance determined at 
step S41, and a numeral value 255 are each decremented by a 
predetermined amount (in Fig. 6, the predetermined amount = 

10 10), and the luminance LH 1 when the maximum luminance 

becomes larger than the numeral value is determined. More 
specifically, in this instance, the numerical value starts 
at 255 and decrements in the manner 245, 235, 225, ... , and 
the numerical value is compared with the maximum luminance 

15 every time the numerical value is decremented (in Fig. 6, 

LH 1 ultimately becomes 245) . Then, a region including 
pixels of a predetermined percentage (in Fig. 6, 1% of the 
number of all pixels) is determined. The minimum luminance 
of this region is assumed as a highlight point LH (in Fig. 6, 

20 LH = 234) . Then, average luminances (RH, GH, and BH) of R, 

G, and B colors within this region (the region where the 
luminance thereof is in a range of LH to LH 1 , both inclusive, 
is calculated by the following expressions. 
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ur , LM' 

RH=£llistR[m]/£HistL[m3 

ia=LH efIJI 

CH=£HistG[ai]/ £ HistL[m] 

n=LH m=LH 
LH' t Ui' 

BH=£HistBW/ SHistLU 

efLH dfLH 

[0080] At step S43, the minimum luminance of the image is 
determined from the histogram HistL stored in the histogram 
holding section 5. In the histogram shown in Fig. 6, the 
5 minimum luminance is 6. 

[0081] At step S44, the minimum luminance determined at 
step S43, and a numeral value 0 are each incremented by a 
predetermined amount (in Fig. 6, the predetermined amount = 
10), and the luminance LS 1 when the minimum luminance 

10 becomes smaller than the numeral value is determined (in Fig. 

6, LS 1 ultimately becomes 10). Then, a region including 
pixels of a predetermined percentage (in Fig. 6, 1% of the 
number of all pixels) is determined. The maximum luminance 
of this region is assumed as a shadow point LS (in Fig. 6, 

15 LS = 22) . Then, the average luminances (RS, GS, and BS) of 

R, G, and B colors within this region (the region where the 
luminance is in a range of LS 1 to LS, both inclusive, is 
calculated by the following expressions. 



RS=£HistR[m]/ £HistL[m] 

a=L5' 3FL5' 

GS=£HistC[m]/ £ Hi stLCin] 

BS=£HistB[m]/ £HistL[m] 
efLS' sfLS' 

[0082] At step S45, lookup tables LUTR, LUTG, and LUTB 
corresponding to R, G, and B colors, respectively, are 
created from the determined RH, GH, BH, RS, GS, and BS . Fig 
8A shows examples of the created lookup tables. A more 
detailed description thereof is given later. Then, at step 
S4 6, the average luminance of the image is determined from 
the Histogram HistL stored in the histogram holding section 
5. A lookup table LUTL for exposure correction is created 
based on the determined average luminance value. Fig. 8B 
shows an example of a created lookup table. A more detailed 
description thereof is made later. 

[0083] At step S47, the lookup tables LUTR, LUTG, and 
LUTB; LUTL created in step S4 6; and the lookup tables LUTRDD 
LUTGDD, and LUTBDD stored in advance in the lookup table 
holding section 7 for correcting gradation characteristic 
and color reproducibility, are synthesized according to the 
following expressions. The synthesized lookup tables LUTR2, 
LUTG2, and LUTB2 are stored in the lookup table holding 
section 7. 
[0084] 
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LUTR2 [i] = LUTRDD [LUTL [LUTR [i]]] 

LUTG2 [i] = LUTGDD [LUTL [LUTG [i] ] ] 

LUTB2 [i] = LUTBDD [LUTL [LUTB [i] ] ] 
Here, "i" is a value in a range of 0 to the maximum 
5 luminance, both inclusive. 

Examples of Created lookup Tables 

[0085] Figs. 8A and 8B show examples of the created LUTR, 
LUTG, LUTB, and LUTL. The lookup tables LUTR, LUTG, and 
LUTB shown in Fig. 8A are used for correcting contrast and 

10 color fogging. Here, gamma (y) values (described later) of 

G, B, and R highlights are increased in this order. By 
enhancing G and B relative to R in this manner, for example, 
red fog on an image can be corrected. Concurrently, the 
correction of contrast is also achievable. 

15 [0086] On the other hand, the lookup table LUTL shown in 

Fig. 8B is one for exposure correction, and is created in 
accordance with the average luminance of an image. 
Specifically, when the average luminance is low, there is a 
high probability of the image being underexposed, and 

20 therefore, a table such as to make an overall image brighter 

is created. Conversely, when the average luminance is high, 
there is a high probability of the image being overexposed, 
and therefore, a table such as to make an overall image 
darker is created. 

25 Operations in Image Correcting Section 8 
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[0087] Fig. 12 shows operational process steps in the 
image correcting section 8. These are more specific 
descriptions of step S23 shown in Fig. 4 and step S73 shown 
in Fig. 14. At step S51, one pixel of image data stored in 
5 the image buffer 9 is fetched. As image data, luminances (R, 

G, and B) of RGB colors are stored. 

[0088] At step S52 , based on the lookup tables LUTR2, 
LUTG2, and LUTB2 that have been synthesized at step S47 or 
step S99, the image data fetched from the image buffer 9 is 

10 corrected. The corrected result is stored in the image 

buffer 10 for use in copying. 

R = LUTR2 [R], G = LUTG2 [G] , B = LUTB [B] 
At step S53, it is checked whether all pixels have been 
completed. If so, the proceeding is finished. Otherwise, 

15 the processing advances to step S51. 

[0089] In this embodiment, in order to speed up creation 
processing of lookup tables and image correction processing, 
lookup tables are prepared for each color component. 
Operations in Face Detection Processing Section 12 

20 [0090] Figs. 13 and 24 show operational process steps in 

the face detection processing section 12. These are more 
specific descriptions of step S3 shown in Fig. 2. 
[0091] At step S61, based on the image data stored in the 
image buffer 9 or image buffer 10 for use in copying (for 

25 the latter, the image data is corrected in step S2), regions 
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where the luminance is locally low are marked one after 
another . 

[0092] At step S62, the regions marked in step 61 are 
paired, and it is determined whether the marked pair are 
5 eyes, based on the equality of the sizes of the paired 

regions, difference in luminance therebetween, angle that 
they -form with respect to the horizon (see Fig. 24A) . 
[0093] At step S63, with respect to the paired regions 
determined to be eyes, a rectangular region around the pair 

10 regions is set. Then, an edge in the vicinity of the 

boundary of the rectangular region and color information 
about the inside of the rectangular region are determined. 
The edge is used for determining the profile of a face. The 
color information is used for determining whether the 

15 average values of the R, G, and B values of the rectangular 

region is within a preset skin color region. In this manner, 
the face region is determined by detecting the position and 
size of a face. The face region is given by the above- 
described rectangular region, which is an oblong region of 

20 which the opposite horizontal and vertical sides are each 

parallel to each other (see Fig. 24B (C) ) . 
[0094] At step S64, a mask data such as to mask the 
determined face region is created, and the created mask data 
is stored in the image buffer 9. This mask data is 

25 information for allowing the face region to be identified. 
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Image Correction Processing When Using Mask Data 
[0095] Fig. 14 shows a flow of image correction 
processing when using mask data in this embodiment. 
[0096] At step S71, in the histogram creating section 4, 
5 a histogram is created based on the image data (the image 

data before being subjected to the correction in step S2) 
that has been stored in the image buffer 9, and the created 
result is stored in the histogram holding section 5. 
Operational process steps in the histogram creating section 

10 4 are shown in Fig. 15 and are discussed later. 

[0097] At step S72, in the lookup table creating section 
6, a lookup table is created. Operational process steps in 
the lookup table creating section 6 are shown in Fig. 16, 
and are discussed later. 

15 [0098] At step S73, in the image correcting section 8, 

the image is corrected. Operational process steps in the 
image correcting section 8 are shown in Fig. 12 in a more 
detail . 

20 Histogram Creation Processing When Using Mask Data 

[0099] Fig. 15 shows operational process steps in the 
histogram creating section 4. These are more specific 
descriptions of step S71 shown in Fig. 14. 

[0100] At step 81, one pixel of image data and mask data 
25 are each fetched one after another from the image buffer 9. 



- 24 - 



[0101] At step S82, from the R, G, and B values of the 
image data, the luminance L of a pertinent pixel is 
determined by the following expression. 
[0102] L = (3*R + 6*G + 1*B) /10 
5 At step S33, the histogram stored in the histogram holding 

section 5 is updated. The histogram holding section 5 holds 
a histogram HistL having the luminance L above calculated; 
HistR, HistG, and HistB that store the cumulative luminance 
values of respective R, G, and B colors by luminance L of 

10 the pertinent pixel; and a histogram HistLMsk having a 

luminance L, for a region masked by mask data. Any initial 
state is 0. The updating of histograms is performed by the 
following expressions . 

HistR [L] = HistR [L] + R 

15 HistG [L] = HistG [L] + G 

HistB [L] = HistB [L] + B 
HistL [L] = HistL [L] + 1 
[0103] At step S84, it is checked, based on the mask data, 
whether the pertinent pixel is masked. If so, the 

20 processing advances to step S85. Otherwise, the processing 
advances to step S86. 

[0104] At step S85, a mask region histogram stored in the 
histogram holding section 5 for calculating the exposure 
state of the mask region, is updated. 
25 HistLMsk [L] = HistLMsk [L] + 1 
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[0105] At step S86, it is checked whether all pixels have 
been completed. If so, the processing is finished. 
Otherwise, the processing advances to step S81. 
Operations in Lookup Table Creating Section 6 When Using 
5 Mask Data 

[0106] Fig. 16 shows operations in lookup table creating 
section- 6 when using mask data. These are more specific 
descriptions of step S72 shown in Fig. 14. 

[0107] At step S91, the maximum luminance of the image is 
10 determined from the histogram HistL stored in the histogram 

holding section 5. 

[0108] At step S92, the maximum luminance determined at 
step S91, and a. numeral value 255 are each decremented by a 
predetermined amount, and the luminance LH 1 when the maximum 

15 luminance becomes larger than the numeral value is 

determined. Then, a region including pixels of a 
predetermined percentage is determined. The minimum 
luminance of this region is assumed as a highlight point LH. 
[0109] At step S93, the minimum luminance of the image is 

2 0 determined from the histogram HistL stored in the histogram 

holding section 5. 

[0110] The processing steps S91 to 93 are substantially 
the same as the processing steps S41 to S43 shown in Fig. 7. 
[0111] At step S94, the minimum luminance determined at 
25 step S93, and a numeral value 0 are each incremented by a 
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predetermined amount (in Fig. 6, the predetermined amount = 
10), and the luminance LS 1 when the minimum luminance 
becomes smaller than the numeral value is determined (in Fig. 
6, LS ' ultimately becomes 10). Then, a region including 
5 pixels of a predetermined percentage (in Fig. 6, 1% of the 
number of all pixels) is determined. The maximum luminance 
of this -region is assumed as a shadow point LS (in Fig. 6, 
LS = 22) . Then, at step S95, the average values (RG, GG, 
and BG) of respective R, G, and B luminances of pixels that 

10 are situated within the region where the luminance thereof 

is in a range of LS to LH, both inclusive, and that meet the 

following expressions, are determined. 

[0112] Max (R, G, B) - Min (R, G, B) < Thr 

Here, Max (R, G, B) denotes maximum values of the R, G, and 

15 B values, and Min (R, G, B) denotes minimum values thereof, 

and Thr denotes a threshold value. 

[0113] At step S96, lookup tables LUTR, LUTG, and LUTB 
are created from the LS, LH, RG, GG, and BG that have been 
determined in step 95. Fig. 17A shows examples of the 

20 created lookup tables. A more detailed description thereof 
is given later. The created results are stored in the 
lookup table holding section 7. Also, a Lookup table LUTTmp 
for correcting HistL is created based on LH and LS . 
[0114] At step S97, HistLMsk is corrected by using the 

25 LUTTmp. Thr corrected HistLMsk is used when an exposure 
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correction amount is calculated later. 

[0115] At step S98, the average luminance of the mask 
region is determined from the HistLMsk after subjected to a 
correction. Then, a lookup table LUTLMsk for exposure 
5 correction is created according to a predetermined method. 

The created result is stored in the lookup table holding 
section 7. Fig. 17B shows an example of LUTLMsk. A more 
detailed description thereof is made later. 
[0116] At step S99, the lookup tables LUTR, LUTG, LUTB, 

10 and LUTLMsk that have been created in step S96 are 

synthesized by the following expressions. The synthesized 
lookup tables LUTR2 , LUTG2, and LUTB2 are stored in the 
lookup table holding section 7. 

LUTR2 [i] = LUTLMsk [LUTR [i] ] 

15 LUTG2 [i] = LUTLMsk [LUTG [i] ] 

LUTB2 [i] = LUTLMsk [LUTB [i] ] 
Here, "i" is a value in the range of 0 to the maximum 
luminance, both inclusive. 

20 Examples of Created Lookup Tables 

[0117] Figs. 17A and 17B show examples of the created 
lookup tables. The lookup tables LUTR, LUTG, and LUTB shown 
in Fig. 17A are used for correcting color fogging. Here, 
gamma values of G, B, and R halftones are increased in this 

25 order. By enhancing G and B relative to R in this manner, 
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for example, red fog on an image can be corrected. 
Concurrently, the correction of contrast can also be 
achieved . 

[0118] On the other hand, the lookup table shown in Fig. 
5 17B is a lookup table LUTLMsk for optimally correcting an 

exposure in the mask region. Here, it is also possible to 
set the region of a person to a mask region, and to set 
LUTLMsk so that the exposure of the person can be optimally 
corrected . 

10 

Operations in Filter Processing Section 11 
[0119] Fig. 18 shows operational process steps in the 
filter processing section 11. These are more specific 
descriptions of step S6 shown in Fig. 2. 

15 [0120] At step S101, enlargement/reduction processing for 

the image is performed. A method such as a bicubic method 
or bilinear method is used for this processing. 
[0121] At step S102, edge enhancement processing is 
performed. Operational process steps of the edge 

20 enhancement processing are shown in Fig, 19, and discussed 

later . 

Edge Enhancement Processing 

[0122] Fig. 19 shows a flow of edge enhancement 
25 processing. These are more specific descriptions of step 
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S102 shown in Fig. 18. 

[0123] First, at step Sill, one pixel of R, G, and B 
luminance data is read from the image buffer 10 for use in 
copying . 

5 [0124] At step S112, the read RGB luminance data and a 

threshold value are compared, for each of the R, G, and B 
colors. If all pieces of R, G, and B luminance data are 
lower than the threshold value, or the pertinent pixel is 
one situated at an edge, the processing advances to step 

10 S114. Otherwise, the processing advances to step S113. 

[0125] At step S113, with respect to color component data 
having a luminance value of not lower than the threshold 
value thereof, a filter processing is performed by using an 
edge enhancement filter of 3 x 3. No edge filter processing 

15 is performed with respect to color component data having a 

value lower than the threshold value. 

[0126] At step S114, the R, G, and B luminance data that 
has been processed is written into the image buffer 9. When 
no edge processing is performed, the R, G, and B luminance 
20 data that has been read at step Sill is written into the 

image buffer 9 as it is. At step 115, it is determined 
whether all pixels have been completed. Otherwise, the 
processing advances to step Sill. 

[0127] Meanwhile, as the threshold value used in step 
25 S112, one and the same threshold value may be used with 
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respect to all color components. Alternatively, the 
threshold value may be independently set with respect to 
each of the color components. 

[0128] According to the first embodiment, it is possible 
5 to determine whether the face of a person is included in the 
input image, and to perform a correction in accordance with 
the determination result. Therefore, if included, the face 
of the person can be arranged to be appropriately color- 
reproduced. 

10 

Second Embodiment 

[0129] In the first embodiment, when creating a lookup 
table, as a highlight point and shadow point after 
correction (i.e., HP and SP shown in Fig. 8A) , certain 

15 constants are always used. However, the use of the 

information stored in the photographic information holding 
section 13 allows highlight points and shadow points after 
correction to be variable. Hereinafter, a second embodiment 
of the present invention is described. 

20 [0130] Fig. 25 shows the construction of the main portion 

of an image processing device according to the second 
embodiment of the present invention. 

[0131] Referring to Fig. 25, this image processing device 
comprises an image input section 2, image output section 3, 
25 histogram creating section 4, histogram holding section 5, 
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lookup table creating section 6, lookup table holding 
section 7, image correcting section 8, imager buffer 9, 
image buffer 10 for use in copying, filter processing 
section 11, face detection processing section 12, 
photographic information holding section 13, and correction 
information holding section 15. 

[0132] Other than the lookup table creating section 6 and 
correction information holding section 15, other features of 
the present embodiment are similar to those of the above- 
described embodiment, and are therefore omitted. 

Method for Creating Lookup Tables in Second Embodiment 
[0133] Fig. 26 shows information stored in the correction 
information holding section 15. 

[0134] As shown in Fig. 26, two conditions: the 
usage/non-usage of a flash, and the range of an in-focus 
position, are combined, and in accordance with the 
combination of these two conditions, highlight points and 
shadow points after correction are held. As a result, for 
example, when a flash is used and the focal length is short, 
that is, when there is a high probability that a highlight 
disappears, too much enhancement of contrast can be 
prevented by reducing the value of HP. Conversely, when no 
flash is used and a focal length is long, that is, when 
there is a high probability that the photograph is a scenic 
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one, contrast can be enhanced by increasing the value of HP. 
These HP and SP are put to use by being read from the 
correction information holding section 15 when creating 
lookup tables in the lookup table creating section 6 (i.e., 
5 at steps S45 and S96) . 

Description of Luminance 

[0135] In the above-described embodiment, digital values 
from 0 to 255 are used as luminance data. However, for 
10 example, the maximum value of the data is not limited to 255. 

Furthermore, besides luminance, halftone dot density may 
also be utilized. 

Method for Calculating Luminance Values 
15 [0136] At steps S32 and S82, the luminance value is 

determined by weighted average calculation using weights of 
(R:G:B = 3:6:1). However, the calculating method for 

luminance values is not restricted to this method. For 

example, weights other than the above-described ones may 
20 also be used for weighted average calculation. Furthermore, 

the luminance may be determined from an average value of the 

maximum value and minimum value of each of the R, G, and B 

luminance . 

25 Method for Representing Face Detection Result 
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[0137] In the above-described embodiment, the result of 
face detection, i.e., a mask region, is represented by a 
rectangle. However, the mask region is not limited to a 
rectangle. For example, the result of face detection may be 
5 a result obtained by extracting a profile portion. The 

information allowing the mask region to be identified may be 
information . about the width and height based on the 
coordinate values of the two opposite points of either one 
of the two diagonal lines of the rectangle (oblong region) , 

10 or information about the width and height based on the 

coordinate values of some of the vertexes of the rectangle 
(oblong region) . Moreover, besides rectangle, the 
information allowing the mask region to be identified may be 
a circular region (or elliptical region) nearer to a human 

15 face. In this case, the information allowing the mask 

region to be identified is the central coordinate values and 
the length of the radius of the circular region, or the 
central coordinate values and the length of the diameter 
thereof, or four vortexes of a rectangle circumscribing the 

20 circular region or the elliptical region. Moreover, the 

mask region may be a polygonal region representing the shape 
of a human face. In this case, information allowing the 
polygonal region to be identified is the coordinates of all 
vortexes thereof. 



25 
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Posture of Camera During Photographing, and Image Data 
Stored in Image Buffer 

[0138] In Fig. 23, the postures of camera during 
photographing and image data stored in the image buffer are 
5 each illustrated for four different cases. However, the 

postures of camera and image data stored in the image buffer 
are not restricted to these four kinds. For example, when 
photographing with a camera inclined by 45°, the image may 
be stored in the image buffer with the image inclined by 45° 
10 in the inverse direction. 

Method for Utilizing Image Information 

[0139] In the above-described embodiment, as photographic 
information, only postures of a camera are used. However, 

15 as shown in Fig. 20, the output object profile shown in Fig. 

9 may be switched by using an input device ID such as a 
camera. Also, when attempting to create LUTL for exposure 
correction, it may be created based on an in-focus position 
by using an in-focus position. 

20 Regarding Information Held in Correction Information Holding 

Section 15 

[0140] In the above-described second embodiment, 
highlight points (HPs) and shadow points (SPs) after 
25 correction are held in the correction information holding 
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section 15. However, information to be held in the 
correction information holding section 15 is not limited to 
HPs and SPs alone. For example, methods for exposure 
correction may also be held. As a consequence, when no 
5 flash is used and focal length is long, that is, when there 

is a high probability of an image being underexposed, the 
exposure correction can be set so as to make the brightness 
of an image a little higher than in normal cases for 
optimizing the exposure of the image.. 

10 

Third Embodiment 

[0141] A third embodiment of the present invention will 
be now described. The descriptions hereinafter are made of 
only different parts of the this embodiment from those in 

15 the first and second embodiments. In the first and second 

embodiments, the average luminance of a face region is 
calculated, and lookup tables are created based on 
predetermined conversion tables. However, in this 
embodiment, another system for creating lookup tables for 

20 exposure is explained. 

[0142] In general, the desirable brightness of a face is 
regarded as approximately 60 to 80% of the maximum luminance. 
In the input image, when the average luminance of a detected 
face region is not within this range (the range of an ideal 

25 color) , by performing a correction such as to bring the 
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average luminance near to this range, a correction such as 
to convert the input image into desirable color information 
(i.e., correction such as to reproduce a desirable human 
skin color) can be achieved. In the descriptions 
5 hereinafter, the desirable brightness of a face is treated 

as approximately 60 to 80% of the maximum luminance. 
However, this value is merely an example. Other values may 
also be adopted. 

[0143] Fig. 27 is a flowchart showing a processing flow 

10 of the creation of the lookup table for exposure correction. 

First, at step S2701, a representative luminance 
(representative color information) of the detected face 
region is calculated. In this embodiment, as the 
representative luminance, average pixel value (average 

15 luminance) of the luminance values of pixels included in the 

face region, are used. Therefore, in the same manner as in 
the first and second embodiments, the average luminance 
Lface of a mask region corresponding to the face region is 
determined, and this average luminance Lface is assumed as 

20 the representative luminance of the face region. 

[0144] A possible modified method for determining the 
representative luminance is to determine a central value or 
mode value out of the histogram HistLMsk in a mask region 
that is the same as the one described in the first 

25 embodiment, and to assume this value as the representative 
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luminance Lface. By establishing an arrangement such that, 
through a user interface of the present device, the user can 
specify which to select as the representative luminance from 
among the central value, mode value, and average value, a 
5 color correction to suit preference of the user can be 

implemented. 

[0145] At the- next step S2702, a target luminance Ltarget 
is calculated. The target luminance Ltarget is calculated 
using the following expression (1) . 

Ltarget n Lface+ (LidealMin~Lface)/A . (Lface'ir LidealMin) 
Ltarget=LFace (Li deal llii n ^Lf ac e £ Li deal Max) 

Ltarget=LFace- (LfaceriidealMax ) /A (Li dealUax ^Lf act 1 ) • - Expression (1 ) 

10 

[0146] In the above expressions, LidealMin and LidealMax, 
respectively, are an upper limit value and lower limit value 
of the desirable luminance range of a face. In this 
embodiment, the LidealMin and LidealMax values, respectively, 

15 are predetermined to be 60% of the maximum luminance (i.e., 

255 x 0.6 = 153) and 80% thereof (i.e., 255 x 0.8 = 204). 
"A" is a parameter for controlling a correction amount, and 
is a number of not less than 1. In the present experiments, 
satisfactory results were obtained by setting "A" to be on 

20 the order of 2 to 4 . 

[0147] Performing a correction such as to bring the 
above-described representative luminance (color information) 
of a mask region near to the ideal color range, comprises 
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the following steps: (1) out of the color information value 
(defined as CI) of the representative color in the region 
calculated from all or some of the mask region, and color 
information values included in the ideal color range, 
5 determining a color information value (defined as C2) that 
is situated at the shortest distance from the color 
information value CI of the representative color in a color 
space defining the above-described color information; (2) 
determining a color information value (defined as C3) that 

10 internally divides the line connecting the color information 

values CI and C2; and thereby, (3) correcting the input 
image so that the color information value CI becomes the 
color information value C3. Here, the internal ratio used 
is a predetermined value. 

15 [0148] Fig. 28 is a diagram showing the relationship 

among the representative luminance, target luminance, and 
desirable luminance range of a face. In Fig. 28, the value 
of "A" is set to 3. Lface 1 and Ltarget 1 are examples in 
which the respective representative luminances are lower 

20 than the lower limit of the desirable luminance range of a 

face; and Lface 2 and Ltarget 2 are examples in which the 
respective representative luminances are within the 
desirable luminance range of a face; and Lface 3 and Ltarget 
3 are examples in which the respective representative 

25 luminances are higher than the higher limit of the desirable 
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luminance range of a face. 

[0149] At step S2703, the gamma (y) value is calculated 
from the representative luminance Lface and target luminance 
Ltarget of the face region. The y value is calculated by 
5 using the following expressions. First, the expression (2) 

for y correction is transformed into the expression (3) . 
Then, in the expression (3), Lface is substituted for x, and 
Ltarget is substituted for y, and thereby a y value is 
determined . 

y-255 X ^ T^rjr j * . . Expression (2) 

10 

(Here, x is a value before correction; y is a value after 
correction; and x and y are each values in the range of 0 to 
the maximum luminance LMAX, both inclusive.) 




■ • Expression (3) 



15 Lastly, at step S2704, the y value determined in step S2703 

is substituted into the expression (2) . While varying the 
value of x from 0 to 255, the value of y corresponding to 
the value of x is calculated. The calculation results are 
stored as a lookup table LUTLMsk, in the lookup table 

20 holding section 7, as in the case of the first embodiment. 

[0150] As described above, a correction such as to bring 
color information near to the desirable color reproduction 
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range is performed. Specifically, in the y correction 
curve: y = LMAX x (x/LMAX) 1/Y (where, 0 < x < LMAX, and 0 < y 
< LMAX) , the y value that passes through (x, y) = (CI, C3) is 
calculated, and the correction of the input image is 
5 performed by using the determined curve. 

[0151] In the first embodiment, the lookup table for 
exposure correction, the lookup tables LUTR, LUTG and LUTB 
for color fogging correction, as well as the calculated 
lookup tables LUTRDD, LUTGDD, and LUTBDD for color 

10 reproducibility correction, are synthesized, and these 

results are applied to R, G, and B values of pixels of the 
input image. However, the calculation and synthesis of 
these lookup tables are not indispensable for the correction 
system according to this embodiment. 

15 [0152] When the calculation and synthesis of these lookup 

tables are not performed, LUTR2, LUTG2, and LUTB2 of the 
first embodiment are equal to LUTLMsk. 

[0153] A modified method for color correction can also be 
utilized in which pixels of the input image are converted 

20 from R, G, and B image data into, for example, YCbCr image 

data, and after its Y has been subjected to a correction by 
LUTLMsk, the converted data is again returned to R, G, and B 
image data. Meanwhile, in the above-described embodiments, 
the y correction is used as a luminance correction method, 

25 but other luminance correction methods may also be employed. 
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[0154] As described above, according to the correction 
system of this embodiment, it is possible to perform image 
correction to match the desirable brightness of a face. 

5 Other Embodiments 

[0155] The present invention may be applied to either a 
system comprising a- plurality of apparatuses (e.g., a host 
computer, interface unit, reader, printer, etc.)', or an 
apparatus comprising a single device (e.g., a copying 

10 machine, facsimile, etc) . 

[0156] The present invention also incorporates a case 
where, with a view toward achieving the functions of the 
above-described embodiments, program codes of software for 
implementing the functions of the above-described 

15 embodiments are supplied to a computer in an apparatus 

connected to various devices or a system in order to actuate 
the various devices, and the computer (CPU or MPU) of the 
system or apparatus is caused to actuate the aforementioned 
various devices in accordance with the stored program. 

20 [0157] In this case, because the program codes themselves 

realizes the functions of the above-described embodiments, 
the program codes themselves, and means for supplying the 
program codes to the computer, for example, a storage medium 
storing such program codes, constitute the present invention. 

25 [0158] The storage medium storing such program codes may 
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be, for example, a floppy® disk, hard disk, optical disk, 
magneto-optical disk, CD-ROM, magnetic tape, nonvolatile 
memory card, ROM. 

[0159] Such program codes are incorporated in the present 
5 invention in not only a case where the functions of the 

above-mentioned embodiment are implemented by the computer 
executing the supplied program codes, but also in a case 
where the functions of the above-mentioned embodiment are 
realized by the program codes cooperating with an operating 
10 system (OS) operating on the computer, or other application 

soft or the like. 

[0160] Furthermore, the present invention incorporates a 
case where the supplied program codes are stored in a memory 
provided for a function expanding board inserted into the 

15 computer or a function expanding unit connected to the 

computer, and thereafter, based on instructions of the 
program codes, a CPU or the like provided for the function 
expanding board or function expanding unit executes some or 
all of the actual processing operations, thereby 

20 implementing the functions of the above-described 

embodiments through these processing operations. 
[0161] As many apparently widely different embodiments of 
the present invention can be made without departing from the 
spirit and scope thereof, it is to be understood that the 

25 invention is not limited to the specific embodiments thereof 
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